Skip to content

feat(realtime): add opt-in Worker Realtime WebSocket transport - #62

Merged
ALikesToCode merged 1 commit into
integrate/gateway-waves-1-5from
feat/realtime
Oct 9, 2026
Merged

ALikesToCode merged 1 commit into
integrate/gateway-waves-1-5from
feat/realtime

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds an opt-in Realtime WebSocket transport served entirely in the Worker. It bridges authenticated OpenAI-compatible sockets to an approved upstream and never wakes the Container. With REALTIME_ENABLED unset, empty, false or malformed (the default is off), handleRealtimeRequest returns null before any other work, and every route, including /v1/realtime, keeps its existing handling byte for byte.

  • Upstreams:
    • REALTIME_PROVIDERS_JSON maps an exact provider:model to a complete wss:// URL on a public DNS host, port 443. IP literals, local or internal names, userinfo, fragments and credential-like query fields are rejected.
    • Client URLs, query fields, hosts and headers never pick an upstream. Redirects are refused.
    • The provider key comes from the Worker's existing provider key binding and never appears in the mapping. The only caller header forwarded is the exact OpenAI-Beta: realtime=v1 version header. Upstream headers are not returned.
  • Connecting: GET /v1/realtime?model=provider:model with Upgrade: websocket and a gateway key in Authorization: Bearer or X-MultiLLM-API-Key.
    • Accepted keys: the bootstrap key, D1 dashboard keys and D1 integration keys. Dashboard model grants, expiry, revocation and IP controls apply, and both chat and audio scopes are required.
    • Keys stored only in the Container can't open sockets.
    • With the flag on: a bad upgrade gets 426, a missing or invalid key 401, a missing scope or model grant 403, and any missing storage, migration, price, ceiling or admission authority 503 before dialling. A full session or budget limit gets 429 and an upstream handshake failure 502, with no retry. All errors are JSON with no-store.
  • Tickets: optional POST /v1/realtime/client_secrets returns a gateway-scoped ticket, never a provider secret.
    • The ticket is HMAC-signed with a key derived from JWT_SECRET, lasts 60 seconds and is bound to one model and the issuing credential.
    • It is consumed atomically once, and a key may hold at most 16 outstanding tickets.
    • A ticket can't issue another ticket. It is presented as the Bearer token.
  • Limits:
    • At most 2 concurrent sockets per key, admitted atomically in D1.
    • The existing admission lease is held for the session's lifetime.
    • 900-second TTL, 30-second idle limit and a 1 MiB frame limit.
    • Policy violations close with 1008. Upstream error events are forwarded verbatim, then both sides close with 1011.
    • An upstream frame containing the provider key is withheld and the session closes with 1008. Closing either side closes the other once.
  • Cost:
    • REALTIME_PRICING_JSON gives per-model prices per million audio/text tokens and a provider_session_limit_usd. That limit must be enforced by the provider account or session, and must not exceed REALTIME_SESSION_CAP_USD.
    • The whole ceiling is reserved through the existing durable usage reservations before dialling. Key daily or monthly budgets apply, otherwise a positive REALTIME_DAILY_BUDGET_USD is required.
    • Measured usage settles through the content-free usage ledger. Partial, inconsistent or interrupted usage keeps an unknown hold instead of inventing a zero bill.
  • Privacy: no prompt, transcript, audio or event content is stored or logged.

Details are in docs/realtime.md.

Deviation from the brief

The brief asked for a monetary cap the gateway enforces. Usage events arrive after the upstream has already spent the money, so the gateway can't guarantee a cap on its own. Admission therefore requires a provider-enforced session ceiling no greater than the gateway cap, and refuses the session when there is none.

Limits to know

  • Browsers: tickets are presented in the Authorization header. Browsers can't set headers on a WebSocket, and credentials in URLs or subprotocols are refused. So a plain browser client can't connect yet; server-side clients and native apps can.
  • Not covered at the edge: Realtime frames don't pass through the secret firewall, and sessions aren't counted against edge rate limits. Sessions are bounded by the two-socket limit, the admission lease and the reserved budget.
  • Testing: verification uses fake WebSocketPair and upstream sockets only. No live provider socket, billing or protocol compatibility has been tested.

Wiring

  • cloudflare-worker.mjs:fetch gets one call, handleRealtimeRequest, right after the health check. A 101 response is returned as is; other responses get the existing CORS headers.
  • worker/gateway-extensions.mjs re-exports the handler, and worker/api-paths.mjs adds isRealtimeRequestPath.
  • wrangler.jsonc is unchanged: no new bindings, Durable Object classes or vars.

Shared registrations

  • tests/test_realtime_worker.mjs in test:worker.
  • worker/realtime.mjs, worker/realtime-session.mjs and worker/realtime-metering.mjs in the edge module inventory.
  • REALTIME_ENABLED and REALTIME_PROVIDERS_JSON in the Container env keys.
  • realtime_tickets and realtime_sessions in REQUIRED_D1_TABLES.

No Flask routes or browser-readable headers.

Tests

tests/test_realtime_worker.mjs covers:

  • flag off, empty and malformed leaving every route unchanged, and flag on never reaching the Container;

  • the URL allowlist and that caller headers never reach upstream;

  • the 426, 401 and 403 responses, scopes, model grants and key controls;

  • ticket signing, expiry, single use, model binding, credential rotation and the no-reissue rule;

  • the 2-session limit, admission lease hold and release, TTL, idle and frame limits;

  • close-code coupling, upstream error forwarding and provider-key echo withholding;

  • price, ceiling and budget refusals before dialling;

  • measured settlement and unknown holds for partial or interrupted usage.

  • Full Python suite: 4,456 passed, 2 skipped. ruff, mypy and bandit are clean.

  • npm run test:worker: 1,656 passed. npm run test:integration: 7 passed.

  • cf:migrations:check reports 29 contiguous, additive migrations. test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.

  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

Apply the migration before enabling: npx wrangler d1 migrations apply multillm-intelligence --remote.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 6. It is based on integrate/gateway-waves-1-5, the Wave 5 integration branch (#60), and it is the base of the Wave 6 migration stack (0029). The README line comes with the Wave 6 integration PR.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8fcac456-6e4a-4162-b189-b5b3c5624ae0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy c3712d1 Oct 09 2026, 05:11 AM

@ALikesToCode
ALikesToCode merged commit f0ceaf3 into integrate/gateway-waves-1-5 Oct 9, 2026
4 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant